Systems, methods and media for dynamically shaped tensors using liquid types

ABSTRACT

Systems, methods, and processor readable media are described for verifying software. A liquid type system is used by a programming language to allow source code to define tensor variables with dimensionality and/or shape dynamically defined at runtime. The dimensionality and shape of a tensor variable invoked in the source code, as well as the data type of the constituent elements of such a tensor variable, may be defined by a static type that may be verified at compile time.

TECHNICAL FIELD

The present application relates to software source code verification andcompilation, and in particular to systems, methods and media forrepresenting dynamically-shaped tensors using a liquid type system.

BACKGROUND

Machine learning DSLs (domain specific languages) are specializedcomputer programming languages used for developing machinelearning-based software systems. A variable defined by a programminglanguage typically has a fixed data type, typically indicating a set offixed characteristics of a variable of a given type. As used herein,“type” and “data type” shall both refer to a data type defined by aprogramming language. Examples of common data types include integers,floating point numbers, and arrays.

Machine learning algorithms tend to make extensive use of tensors, i.e.multi-dimensional matrices of values, which may represent structuressuch as convolution kernels and activation maps in the context ofmachine learning. As used herein, the term “tensor” may refer to amulti-dimensional array of values, including an array of arrays.

Software verification refers to a process whereby software source codeis automatically checked for compliance with standards imposed by theprogramming language, such as type safety. Verification may be carriedout by a compiler or other software tool. (As used herein,“verification” shall refer to software verification.) When a variable isinvoked for a specific purpose in the source code, such as by beingpassed as a parameter to a function, the compiler may verify that thevariable being invoked is defined in the source code as being of acorrect type for use as the given parameter by the given function. Ifthe type of the variable is incorrect, the source code is deemed to beinvalid by the compiler, and the compiler may refuse to compile the codeand/or alert the user to the presence of a compilation or verificationerror.

In existing programming languages used for machine learning, includingmachine learning DSLs, tensors may be represented by variables of anexisting fixed type, such as an array with a defined number ofdimensions, or by dynamic data types defined within the source code,e.g. by a software library, based on existing fixed data types (e.g.,arrays of arrays with dimensionality defined by various parameterspassed to functions defined by the library). In either case, a compilerperforming software verification may only verify the type safety oftensors at the level of the fixed, static data types (e.g. array)defined by the programming language. A software library defining adynamic data type must include functions for checking type safety, e.g.,to verify that the dimensionality of a tensor passed as a parameter to afunction matches the required dimensionality for that parameter. Thisnot only adds to the complexity of the source code used to define anduse tensors, but it also adds additional computation at runtime asopposed to compile time, slowing the execution of the compiled sourcecode.

Thus, the dimensionality (i.e., how many dimensions the tensor has)and/or shape of a tensor (i.e., the size of the tensor in eachdimension) may be defined or verified in the source code by callingfunctions defined by a software library. However, in some cases, thedimensionality and/or shape of the tensor must be defined in the sourcecode itself, instead of allowing the compiled program to dynamicallydefine these aspects of the tensor at runtime based on data inputs.Especially in the context of Natural Language Processing (NLP), manysoftware applications require allocations and operations on tensorshaving dimensionalities or shapes that are unknown at compile time, butare fixed at runtime.

In the context of machine learning software, such as in natural languageprocessing model, an inference model trained using machine learningtakes a tensor as an input. As the shape and/or other properties of thetensor may be dynamic and unknown, it may be a challenge to representthe tensor while programming the natural language processing model,which can lead to inaccurate output. Furthermore, when checking thevalidity of the output, a plurality of manual checking functions must beadded to the source code to ensure the validity of the output.

For example, the input length of a BiDAF (Bidirectional Attention Flow)tensor used in natural language processing may not be known untilruntime. The BiDAF length determines a fixed parameter among multiplelayers in an NLP model. Using existing techniques, a software developerwill have to either waste time on trivial bugs caused by accidentalmisuse of the BiDAF tensor, or manually add checks and shape calculationthroughout the source code, potentially resulting in code smell andsacrificing optimization opportunities that could result from exposingof tensor size information to the compiler.

To address these issues of coding complexity, computational efficiency,and flexibility it may therefore be advantageous to provide programminglanguages, compilers, and verification techniques that define a statictype (i.e., a data type defined at the level of the programminglanguage) for tensors of a shape and/or dimensionality that can bedynamically defined at runtime or that otherwise overcome one or more ofthe aforementioned disadvantages of the prior art.

SUMMARY

The present disclosure provides systems, methods, and processor readablemedia for verifying software. In some embodiments, a liquid type systemmay be used by a programming language to allow source code to definetensor variables with dimensionality and/or shape defined at runtime. Insome embodiments, the dimensionality and shape of a tensor variableinvoked in the source code, as well as the data type of the constituentelements of such a tensor variable, may be defined by a static type thatmay be verified at compile time.

Functional programming languages may be used to verify the validity ofprogram source code, which can help to improve programmer productivityand software reliability. For example, a liquid type system, also calleda logically qualified data type system, is a programming language whichhas been applied to infer dependent types (i.e. types whose definitiondepends on a value, such as an array type defined by a length parameter)precisely enough to prove a variety of validity properties. A liquidtype system provides a means to specify and automatically verifysemantic properties of the source code. A liquid type system may defineconstraints on dependent types where the source code is decidable, whichprovides automatic verification of source code with low requirements fortype annotation.

A liquid type system may provides benefits for dependent types, such asstatic verification of critical properties and the elimination ofexpensive run-time checks with minimal manual annotation.

In some aspects, the present disclosure describes a method for verifyingsource code of a computer program written in a programming language. Themethod comprises receiving the source code, identifying a reference to atensor variable in the source code, and compiling the source code togenerate a constraint. The constraint specifies that the tensor variableis represented by a first sized array of a sized array data type. Thesized array data type is defined by the programming language such that asized array of the sized array data type comprises one or more arrayelements, an associated type parameter that specifies a type for thearray elements, and an associated size parameter that specifies a sizeof the sized array. The type parameter is selected from of a pluralityof types, the plurality of types including the sized array data type.The size corresponds to the number of array elements. The method furthercomprises generating one or more constraint solutions based on theconstraint to verify the source code.

In some examples, the method further comprises identifying a call to aprimitive function in the source code, and identifying the tensorvariable invoked as a parameter of the primitive function in the sourcecode. The tensor variable is specified by the source code to include ashape parameter that specifies a plurality of dimensional sizes of thetensor variable, a plurality of components, and a type parameter thatspecifies a type for the components. The number of componentscorresponds to the product of the dimensional sizes. The size of thefirst sized array corresponds to one of the dimensional sizes of thetensor variable. The associated type parameter of the first sized arrayis a sized array type. The array elements of the first sized array are afirst recursively nested set of sized arrays, such that the size of thefirst sized array and the sizes of each recursively nested set of sizedarrays within the first sized array collectively correspond to thedimensional sizes of the tensor variable. The constraint furtherspecifies that the first sized array is input to the primitive functionto invoke the primitive function.

In some examples, the constraint includes one or more well-formednessjudgement statements. Generating the constraint further comprisesdefining a plurality of well-formedness judgement constraints that statea type of the first sized array. Generating the constraint furthercomprises generating well-formedness judgement statements based on theplurality of defined well-formedness judgement constraints until anassociated type parameter of a recursively nested sized array of thefirst sized array corresponds to the type parameter of the tensorvariable.

In some examples, the constraint includes one or more subtypingjudgement statements. Generating the constraint for the input tensorfurther comprises defining a plurality of subtype judgement constraintsthat state a type of the first sized array. The type of the first sizedarray is a subtype of a predefined sized array format. The predefinedsized array format comprises a sized array having a predefined typeparameter value and a predefined size parameter value for itself andeach of its recursively nested sized arrays. Generating the constraintfor the input tensor further comprises generating the subtypingjudgements based on the plurality of defined subtype judgementconstraints until an associated type parameter of a recursively nestedsized array of the first sized array corresponds to the type parameterof the tensor variable.

In some examples, the programming language defines a dependent pairtype. A dependent pair of the dependent pair type comprises a firstcomponent having a value, and a second component having a type, the typebeing based on the value of the first component, and the value of theshape parameter of the tensor variable is defined by a second componenttype of a dependent pair.

In some examples, the method further comprises registering the primitivefunction and the tensor variable by representing the dimensional sizesof the tensor variable as implicit parameters.

In some examples, the method further comprises, in response todetermining that the source code is uncertain as a result ofverification, modifying the source code by inserting an assertionfunction into the source code to automatically verify the modifiedsource code during runtime.

In some examples, generating constraint solutions further comprisesusing a satisfiability modulo theories (SMT) solver to infer a validtyping for the source code, determining whether the generated constraintsatisfies the inferred valid typing.

In some examples, the method further comprises outputting a result withrespect to a verified validity of the program source code.

In some aspects, the present disclosure describes a computer-readablemedium having instructions tangibly stored thereon, wherein theinstructions, when executed by a processing unit, causes the processingunit to perform the method steps above.

BRIEF DESCRIPTION OF THE DRAWINGS

Reference will now be made, by way of example, to the accompanyingfigures which show example embodiments of the present application, andin which:

FIG. 1 (prior art) is an high-level comparison of three broadly-definedtype systems used for verifying source code, showing how each typesystem would represent type annotation for a function allocating atwo-dimensional tensor with a size defined by an input.

FIG. 2 (prior art) illustrates a flowchart of a method for verifyingsource code.

FIG. 3 illustrates a flowchart of a modified version of the method ofFIG. 2, utilizing a modified liquid type system according to an exampleembodiment.

FIG. 4 illustrates a flowchart of a method for verifying and compilingsource code that may be carried out by a software verification system,according to an example embodiment.

FIG. 5A shows an example of registering a type declaration of aprimitive function matmul using the method of FIG. 4.

FIG. 5B shows an example of source code of a computer program containinga call to the primitive function matmul, which may be verified andcompiled by the method of FIG. 4.

FIG. 5C shows an example tensor variable represented using the method ofFIG. 4.

FIG. 6A shows well-formedness judgement constraints defined by themethod of FIG. 4.

FIG. 6B shows further well-formedness judgement constraints defined bythe method of FIG. 4.

FIGS. 7A and 7B show subtyping judgement constraints defined by themethod of FIG. 4.

FIG. 8 illustrates an example of generating constraints based on thesource code of the computer program of FIG. 5B.

FIG. 9 illustrates an example of a dependent pair used to define a typeof a sized array according to an example embodiment.

FIG. 10 illustrates an alternative example of registering a primitivefunction of the source code of FIG. 5B.

FIG. 11 illustrates an example of modifying the source code of FIG. 5Baccording to an example embodiment.

FIG. 12 is a block diagram illustrating a processing system which maycarry out the methods of FIGS. 3 and 4, according to exampleembodiments.

Like reference numerals are used throughout the Figures to denotesimilar elements and features. While aspects of the invention will bedescribed in conjunction with the illustrated embodiments, it will beunderstood that it is not intended to limit the invention to suchembodiments.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

The present disclosure provides systems, methods, and processor readablemedia for verifying software. In some embodiments, a liquid type systemmay be used by a programming language to allow source code to definetensor variables with dimensionality and/or shape defined at runtime. Insome embodiments, the dimensionality and shape of a tensor variableinvoked in the source code, as well as the data type of the constituentelements of such a tensor variable, may be defined by a static type thatmay be verified at compile time.

To support dynamic shape relations at the type system level, a typesystem must allow a developer to write a function wherein the returntype of the function and the shape of the return type will dynamicallyvary according to the value of an input. The input will typically be ofa fixed type (such as integer type, herein abbreviated to int). A statictype system should preferably achieve a good balance betweenexpressiveness and accessibility without sacrificing safety.

FIG. 1 (prior art) shows three verification systems, which may be usedto detect errors in the source code of a program at compile time. Thethree type systems are, in order of increasing expressiveness anddecreasing accessibility and decidability, a Hindley-Milner (HM) typesystem 102, a liquid type system 104, and a full dependent type system106. Each type system is shown with an example type annotation 102(1),104(1), 106(1) within the given type system 102, 104, 106 for a functionallocating a two-dimensional tensor with a size defined by an input.

A Hindley-Milner (HM) type system 102 is a type system for lambdacalculus with parametric polymorphism. The HM type system 102 has a typeconstraint with a format as denoted by statement 102(1), which does notexpress the shape information of variables at the type level. Therefore,it is challenging for the HM type system 102 to detect errors in theprogram stemming from a shape mismatch.

A full dependent type system 106 is a type system using dependent typesto express all shape information in detail. Expression 106(1) shows thatevery single aspect of shape information is defined for a variable,losing most of the advantages of type inference and requiring extensivetype annotation, make it less accessible.

A liquid type system 104 is a refinement type system wherein logicalpredicates and values are specified with semantic properties. A typeconstraint of the liquid type system may include a statement 104(1)wherein a logical predicate (e.g., len(v)=n) is used. A conventionalliquid type system 104 has a balanced trade-off between expressiveness,accessibility, and decidability, but is unable to precisely performtype-checking and type inference when tensors with multiple differentsets of shape information are invoked in the source code.

Among the three different type systems of FIG. 1, the liquid type system104 may be used as the starting point for embodiments described herein,as it may provide an advantageous balance among the different factorsdiscussed above.

A Tensor type may be provided by the type system as an intrinsic type.This may make type checking easier and provide the compiler with moreinformation about the shapes of the tensors in order to assist withoptimization.

A type and shape annotation mechanism may also be added to the typesystem to let a software developer express complex shape relations thatthe liquid type system 104 itself fails to infer. The type system mayalso add a fall-back strategy that is activated when the liquid typesystem 104 fails to check the correctness of the annotated symbolicrelations of the shape. The fall-back strategy may also be chosen toachieve a balance between expressiveness and accessibility.

Thus, dynamic shape relations may be supported at the type system levelin example embodiments in one or more of three ways: by providing a typesystem permitting functions wherein the return type of the function andthe shape of the return type will dynamically vary according to thevalue of an input; by providing a Tensor type as an intrinsic type; andby providing a type and shape annotation mechanism that can expresscomplex shape relations that the liquid type system fails to infer,potentially including one or more fall-back strategies for when theliquid type system fails.

In some embodiments, the disclosure may provide a liquid type system 104suitable for Machine Learning DSLs. By introducing the concept ofdependent functions, a liquid type system 104 may demonstrateexpressiveness at the type system level: it can express functionsreturning different types and shapes when given arguments of the sametype. By imposing constraints on the type system constraints, a liquidtype system 104 may maintain a high level of decidability with respectto type checking and type inference while remaining relatively easy touse. By finding correspondence between liquid types and logicalstatement, a liquid type system 104 can utilize a satisfiability modulotheories (SMT) solver to speed up its inference, make it practical forindustrial use and checking more invariants about the behavior of theprogram than just shape invariants.

In some embodiments, the disclosure may provide a liquid type system 104supplemented by a “sized array” data type—a type wherein the length ofthe array is part of its type. In some embodiments, the disclosure mayprovide a liquid type system 104 supplemented by a “dependent pair”concept—a pair type wherein the type of the second element can varyaccording to the value of the first element.

A sized array type is a powerful concept due to its simplicity andconciseness, and it is easy to use a sized array type to implement amodel for representing multidimensional tensors at the type systemlevel. In addition, by using the sized array type as the basic buildingblock for tensors instead of directly introducing a tensor type, atensor of tensors is definitionally equivalent to a tensor, resulting inan elegant representation of arbitrary tensors at the type system level.

The dependent pair concept expands the expressiveness of shapeinformation and computation in the type system. Dependent pairs mayallow developers to write source code returning a tensor with acompletely dynamic shape. Source code can be written that directlyreturns a dependent pair, wherein the array (as a second element of thedependent pair) can have shape information defined by the data of thefirst element of the dependent pair.

Some embodiments may use shape annotation. Some such embodiments mayalso require one or more fall-back strategies as described above. Insome embodiments, two fall-back strategies may be employed: first,moving undecided shape checking into runtime, and second, allowingdevelopers to manually write proofs for shape relations that the liquidtype system 104 fails to prove at compile time. The first approachrelies on the SMT solver's ability to precisely categorize the shapeconstraints into three groups: right, wrong and uncertain. Uncertainshape constraints are transformed into dynamic runtime assertions, whichmay be helpful for providing debug information. The second approachrelies on the SMT solver's ability to express equational proofs. SMTonly supports syntactical reasoning; therefore, some embodiments mayprovide an easy-to-use interface for a developer to connect each step ofa proof.

An example use case will now be described for a modified liquid typesystem as described above. The modified liquid type system may use anexisting HM type system 102 as a baseline and add certain operationsimplementing the various liquid type features described above. Theexample use case illustrates a part of the data pipeline forverification of a program, such as an NLP model programmed using machinelearning techniques, using an HM type system 102 in FIG. 2. A modifiedversion incorporating the liquid type system features described above isshown in FIG. 3.

FIG. 2 (prior art) is a flowchart illustrating a conventional method forperforming verification of a program, implemented by a HM type system102, such as a compiler.

At 202, source code for the program is parsed by the HM system 102.

At 204, type-checking and type inference are performed with respect tothe source code. These processes make use of known HM type system 102techniques for type checking and type inference.

At 206, typeclasses are eliminated. In cases wherein a given type isdefined to be an instance of a given class in two different ways (e.g.,two typeclasses), it may be difficult to decide whether the two giventypeclasses overlap. Therefore, the HM type system 102 explicitly addsand removes one of the two given typeclasses to verify whether the twogiven typeclasses overlap.

FIG. 3 is a flowchart illustrating a method to perform verification of aprogram using a modified liquid type system as described above. In themodified liquid type system, tensors are invoked as variables. Themethod 300 includes steps 202-210 as discussed in the example of FIG. 2,and further includes steps 302 and 304, which may enable thetype-checking of the program to be performed with greater accuracy.

At 302, constraints are generated. This step may be implemented by aconstraint generator. The constraint generator may first go through anintermediate representation (IR) tree that is generated in the compilerinternally to represent the source code of the program. Each IR node ofthe IR tree is associated with a type. The constraint generatorgenerates a template for each type. For example, with respect tofunction types for each function call, the constraint generator producesa dependent function type for each function call. With respect totensors, templates for tensors' types are generated with reference toFIGS. 6A-6B and 7A-7B, which will be discussed in detail below. Thejudgement constraints containing templates are also called“well-formedness judgement constraints” and “subtyping judgementconstraints” in the examples of FIGS. 6A-6B and 7A-7B. Once thetemplates for function calls and tensors are generated, well-formednessjudgement statements and subtyping judgement statements are generated,such as constraints 802 in the example of FIG. 8.

At 304, constraint solutions are generated. In some examples, eachgenerated well-formed judgement statement for a dependent function issplit into simple judgement statements for each base type. Interactiveweakening may be performed at step 304(1). A satisfiability modulotheories (SMT) solver may be then be used to check the validity of thesource code at step 304(2). The SMT solver may infer a valid typing forthe source code, and determine whether the generated constraintsatisfies the inferred valid typing. Steps 304(1) and 304(2) may beiterated in alternating sequence one or more times.

FIG. 4 shows an example method 400 of verifying and compiling sourcecode that may be implemented by a verification system, such as acompiler, using a modified liquid type system in accordance with thepresent disclosure.

At 402, type declarations are registered for at least one primitivefunction. The primitive function may be any built-in function orpredetermined function defined at the level of the programming language,such as add or multiply. In examples described herein, the primitivefunction is a matrix multiplication function matmul. The typedeclarations for the primitive function specify at least one tensorvariable as a parameter to the primitive function. In the presentexamples, the primitive function matmul specifies two tensors as itsinput parameters, and the function returns a tensor as its output. Itwill be appreciated that matrix multiplication requires that thedimensions of the operands (i.e. the input parameters) satisfy certaincriteria in order for matrix multiplication to be possible: for example,for two two-dimensional operand matrices, the width of the first operandmust be equal to the height of the second operand. Thus, the tensorspassed to the matmul function must have specific shape characteristicsin order for the function call to be valid. Before continuing with thedescription of FIG. 4, a detailed example of type declarations for aprimitive function will be described with reference to FIG. 5A.

FIG. 5A shows an example of registering a primitive function of theprogram 502. Prior to providing the source code to the compiler,primitive functions of the programming language are registered, andparameters of the primitive functions are defined, such that calls tothose primitive functions and parameters of the primitive functions inthe source code will be verified when the source code is compiled. Inkeeping with the current example, FIG. 5A shows the type declaration forthe matmul primitive function.

In FIG. 5A, each parameter within matmul will be replaced by arespective tensor, for example, Tensor [(m, n), T]. Taking tensor [(m,n), T] as an example, m and n indicate this tensor is a two-dimensionaltensor, m represents a value of rows of the two-dimensional tensor, andn represents a value of columns of the two-dimensional tensor. Tindicates a type of constituents of the two-dimensional tensor.

Returning to FIG. 4, at step 404, the source code is received. Adetailed example of source code will be described with reference to FIG.5B.

FIG. 5B presents an example of source code 502 that may be verified andcompiled using the method 400 of FIG. 4. The source code 502 defines afunction sampleFunction that takes two input parameters: a, an integer(type int), and b, also type int. The source code allocates a tensor tusing the alloc function with the input parameters a and b. The sourcecode 502 then calls primitive function matmul with input parameters tand t at line 502(1).

In the example of FIG. 5B, the matmul function called at line 502(1) isa primitive function. When the matmul function 502(1) is called in thesource code 502, each parameter t within the matmul function 502(1) willbe replaced by a tensor that is declared in a type declaration withreference to FIG. 5A.

It will be appreciated that conventional type systems, such as HM typesystems 102, may not be capable of verifying the source code 502, and inparticular line 502(1), to determine that the matmul function is beinginvoked using input parameters (t,t) that render the function callinvalid (because a 2 by 3 matrix cannot be multiplied by another 2 by 3matrix). Some embodiments described herein may provide a solution tothis problem by defining tensors using a liquid type system such thateven a tensor with a shape and data type for its constituent elementsdefined at runtime may be verified as to its validity as a parameterwithin the source code.

At 406, a reference to a tensor variable in the source code isidentified. In source code 502, for example, there are multiplereferences to tensor variable t.

At 408, a call to a primitive function in the source code is identified.In source code 502, the primitive function matmul is called at line502(1).

At 410, the tensor variable invoked as a parameter of the primitivefunction in the source code is identified. In source code 502, tensorvariable t is invoked as a parameter of the primitive function matmul atline 502(1). The type of the tensor variable (e.g., t) has severalparameters: a shape parameter that specifies a plurality of dimensionalsizes of the tensor variable (namely two dimensional sizes defined byinteger parameters a and b); a plurality of components, the number ofcomponents corresponding to the product of the dimensional sizes (namelythe uninitialized constituent components or elements of the a by btensor defined by the alloc(a,b) function); and a type parameter thatspecifies a type for the components (namely the data type assigned tothe constituent components of the tensor t by the alloc function).

At 412, the source code is traversed and inspected, generating at leastone constraint for the primitive function. In some examples, theconstraints may include well-formedness constraints (also known aswell-formedness judgement statements) and subtyping constraints (alsoknown as subtyping judgement statements). In the judgement statements,each tensor variable (e.g., tensor variable t) within the primitivefunction (e.g., matmul(t,t)) is represented by a first sized array of asized array data type.

The sized array data type allows tensor variables of arbitrary shape tobe further assessed using recursion. The sized array data type isdefined by the liquid type system of the programming language such thata sized array of the sized array data type comprises one or more arrayelements, an associated type parameter that specifies a type for thearray elements, and an associated size parameter that specifies a sizeof the sized array, the size corresponding to the number of arrayelements. What distinguishes the sized array data type from a standardone-dimensional array is that the type parameter may specify “sizedarray” as the data type for the elements of the sized array from aplurality of data types, enabling sized arrays to further assess tensorsof arbitrary shape with elements of arbitrary data types usingrecursion.

A sized array may thus be represented in the format SArray[L, T],wherein L represents a length (i.e. size parameter) of the sized array,and T represents the type (i.e. type parameter) of the constituentelements contained within the sized array. A detailed example of atensor representation by the liquid type system will be described withreference to FIG. 5C.

FIG. 5C shows how to use a tensor format as described herein to presenta detailed example of a tensor 506. For example, the tensor 506 could berepresented in the format Tensor[(2,3),int] wherein 2 indicates thenumber of rows of the tensor 506, 3 indicates the number of columns ofthe tensor 506, and int indicates that the constituent elementscontained in the tensor 506 are of type int, i.e. integers (shown inFIG. 5C as integer value 5).

In an alternate example, the tensor type parameter is float (indicatinga floating point number), and the constituents could be shown havingfloat value 5.0. Using such a format to represent a tensor enablesoperators to acquire thorough information regarding the tensor (e.g.,size, data type of constituent elements), which helps to ensure accuracyof outputs of the operators when the tensor is provided as input to theoperators.

Taking the tensor 506 as an example, in order to allow the tensor formatto be verified by the verification system, the tensor formatTensor[(2,3),int] can be recursively represented by a plurality ofnested sized arrays with the format SArray[L,T] as follows:Tensor[(2,3),int]=SArray[2,SArray[3,int]]. This tensor corresponds totensor t of FIG. 5B if a=2 and b=3.

This representation is constructed as follows. The first sized array(i.e. the base or top-level sized array corresponding to a baserecursion level) is represented as a sized array of size 2, i.e.SArray[2,type], i.e. an array with two rows of an as-yet undefined type.A sized array SArray[3,int] is then used to represent each row of thetensor 506, such that the first sized array can be represented asSArray[2,SArray [3, int]], indicating a matrix with a first dimensionallength of 2 (e.g., rows) and a second dimensional length of 3 (e.g.,columns). Thus, the type parameter for the first sized array isSArray[3,int]. That is, [5 5 5] is one single constituent element of thefirst sized array.

In more general terms, the size of the first sized array (e.g., 2)corresponds to one of the dimensional sizes (e.g., integer a) of thetensor variable (e.g., tensor t). The associated type parameter of thefirst sized array is a sized array type (e.g., SArray[3,int]). The arrayelements of the first sized array are a first recursively nested set ofsized arrays (e.g. instances of type SArray[3,int]), such that the sizeof the first sized array (e.g., 2) and the sizes of each recursivelynested set of sized arrays within the first sized array (e.g., 3)collectively correspond to the dimensional sizes of the tensor variable(e.g., 2 by 3). The constraint further specifies that the first sizedarray (e.g., t=SArray[2,SArray [3, int]]) is input to the primitivefunction (e.g., matmul) to invoke the primitive function.

In some examples, the number of sized arrays used to recursivelyrepresent a tensor depends on the number of size elements included inthe tensor. For example, if there is one single size element included inthe tensor, only one sized array is used to represent the tensor.

Returning to FIG. 4, constraint generation at step 412 may involve oneor more optional steps (optional steps being shown in FIG. 4 in dashedlines). In some embodiments, well-formedness judgement constraints maybe defined at step 414 to allow the generation of well-formednessjudgement statements at step 416, and/or subtyping judgement constraintsmay be defined at step 418 to allow the generation of subtypingjudgement statements at step 420, thereby representing each tensor as aplurality of recursively nested sized arrays, which will be discussed ingreater detail below with reference to FIGS. 6A through 7B.Well-formedness judgment constraints and subtyping judgement constraintsmay enable the liquid type system to define the tensor, as representedby the sized array type format, to be verified and compiled by theverification system with greater accuracy.

At 422, constraint solutions are generated, based on the constraintsgenerated at step 412, to verify (i.e., determine the validity of) thesource code. In some examples, generating constraint solution mayinclude one or more optional sub-steps. In some embodiments, at optionalstep 424, an inferred valid typing is used to determine whether thegenerated constraints satisfy the inferred valid typing. At optionalstep 426, an SMT solver may be used in a fixed point algorithm to infera valid type. The inferred value type is generated and is one of thegeneral types that satisfy the constraints. Steps 424 and 426 may beiterated, with each of the two steps feeding back to the other.

At further optional step 428, in response to determining that the sourcecode is not valid (e.g. at step 422 or 426), the source code may bemodified by inserting an assertion function as a fall-back strategy.

At further optional step 430, the verification system outputs a resultwith respect to the validity of the source code. For example, if thegenerated constraints are satisfiable, the verification system mayoutput a “safe” or “valid” signifier, meaning that the source code 502has been successfully verified. If the generated constraints are notsatisfiable, the verification system may output an “unsafe” or “invalid”signifier, along with a report for the developer identifying type errorsat positions where the invalid constraints are created. This report mayenable the developer to debug errors in the source code 502.

Thus, in some embodiments, tensors with dynamic shape information (e.g.,dynamic number of dimensions, size of each dimension, and data type ofconstituent elements) may be provided as input to operators of aprogram, such as a neural network used as a natural language processingmodel. The tensors may be represented as parameters of primitivefunctions, with the tensors' shape information providing thoroughinformation for verifying the source code of the program. Furthermore,specific constraints may be generated with respect to the tensors suchthat the verification system can compile the tensors provided asparameters to the primitive function, enabling validity result to begenerated with greater reliability and accuracy. In addition, theefficiency of source code debugging may be boosted significantly.

In some embodiments described above, no type defining the tensor formatTensor[(m,n),type] may yet be recognizable by the verification system.It may be challenging to compile and invoke primitive functions whereinthe tensor format Tensor[(m,n),type] is used. Therefore, when theconstraints are generated, such as at step 412 of method 400, aplurality of well-formedness judgement constraints may be generated atstep 414, and a plurality of subtyping judgement constraints may begenerated at step 418. The well-formedness judgement constraints andsubtyping judgement constraints may be specifically defined for thedefined tensor format Tensor[(m,n),type] in order to enable the tensordefined by size elements (m,n) and contained type to be compiled in theverification system. An example of defining a plurality ofwell-formedness judgement constraints and plurality of subtypingjudgement constraints for the defined tensor format Tensor[(m,n),type]will be described in greater detail with reference to FIGS. 6A through7B.

FIG. 6A shows a general syntax, constituting a well-formedness judgementconstraint 602, for a sized array that is used to represent a tensor.The constraint 602 states that a dependent type of j (e.g., j is atensor) is well-formed, and j's type is defined by a sized arraySArray[i,T]. Thus, the tensor j is a one-dimensional array, and a singlesized array SArray[i,T] is used to define the tensor j. To represent amulti-dimensional tensor, such as two-dimensional tensorTensor[(2,3),int]), the syntax will be defined with reference to FIG.6B.

FIG. 6B shows a syntax wherein two sized arrays are used to representthe tensor Tensor[(2,3),int]. A well-formedness judgement constraint 604includes first, second, and third clauses 604(1)-(3). First clause604(1) states that a dependent type of j (e.g., j is typeTensor[(2,3),int]) is well-formed, and j's type is defined by a sizedarray SArray[2,SArray[3,int]].

Clause 604(2) states that a dependent type of y is well-formed and y'stype is defined by a sized array SArray[3,int]. As a type of the sizedarray SArray[3,int] is not yet recognizable by the verification system,the sized array SArray[3,int] has to be further assessed using recursionuntil a type int in clause 604(3) can be recognized by the verificationsystem.

Thus, a constraint may include one or more well-formedness judgementconstraints (e.g., 604) that state a type of the first sized array(e.g., j=SArray[2,SArray[3,int]]). Well-formedness judgement statementsare generated based on the plurality of defined well-formednessjudgement constraints until an associated type parameter of arecursively nested sized array (e.g., type parameter int ofy=SArray[3,int]) of the first sized array (e.g., j) corresponds to thetype parameter of the tensor variable (e.g. the type parameter of tensorvariable Tensor[(2,3),int], i.e. x=int).

Therefore, in order to ensure each tensor can be compiled by theverification system, the plurality of well-formedness judgementconstraints are defined to use a plurality of sized arrays torecursively represent a tensor or a dependent type of the tensor until atype (e.g., int) used to represent the elements of tensor can berecognized by the verification system.

The defined well-formedness judgement constraints, such aswell-formedness judgement constraints 602 or 604, enable well-formednessconstraints (also known as well-formedness judgements or well-formednessjudgement statements) regarding the tensors invoked by the source code502 to be generated (e.g., at step 416 of method 400) in a manner thatthe verification system can recognize. This may help to improve theaccuracy of verification of sources code 502 and allow tensors definedwith shape information to be verified at compile time when invoked insource code 502.

FIGS. 7A and 7B illustrate an example of defining subtyping judgementconstraints in accordance with example embodiments, which may beimplemented at step 418 of method 400.

FIG. 7A presents subtyping judgement constraints 702 for aone-dimensional tensor. For example, if the one-dimensional tensor isrepresented by a sized array SArray[i,T], subtyping judgementconstraints 702 define that a dependent type of the sized arraySArray[i] is a subtype of a sized array SArray[j,int]. In some examples,the sized array SArray[j,int] may be a result that is generated fromanother program or stored from another program.

FIG. 7B demonstrates subtyping judgement constraints 704 for atwo-dimensional tensor. The two-dimensional tensor is shown as beingrepresented by a sized array SArray[i,SArray[j,int]] (e.g.,SArray[2,SArray[3,int]]). The subtyping judgement constraints 704 definehow to recursively construct the sized array SArray[i,SArray[j,int]]until a type within a final sized array can be recognized by theverification system.

Clause 704(1) states that a dependent type of the sized arraySArray[i,SArray[j,int]] is a subtype of a SArray[k,SArray[w,int]. As atype with the SArray[k,SArray[w,int] is not yet recognizable by theverification system, SArray[k,SArray[w,int] has to be further defined.

Clause 704(2) states that a dependent type of a sized arraySArray[i,SArray[j,int]] is a subtype of a SArray[w,int]. As a type withthe type parameter SArray[w,int] is not yet recognizable by theverification system, SArray[w,int] has to be further defined.

Clause 704(3) states that a dependent type of int is a subtype of a typeint. The constraints 704 do not stop assessing nested sized arrays usingrecursion until the type int of a final sized array can be recognised bythe verification system.

The defined subtyping judgement constraints (e.g., 704) enable subtypingconstraints (also known as subtyping judgements or subtyping judgementstatements) regarding the tensors applied in the source code 502 to begenerated accurately (e.g., at step 420 of method 400), leading thetensors defined with shape information to be accepted and recognized bythe verification system. The reliability of the validity result ofsource code may be increased accordingly.

In some examples, the well-formedness judgement constraints and thesubtyping judgement constraints are defined to collectively improve theaccuracy and reliability of validity results of verification of sourcecode.

Once the well-formedness judgement constraints and the subtypingjudgement constraints are defined, each statement of the source codewill be converted to one or more well-formedness judgement statements(at step 416) and one or more subtyping judgement statements (at step420) during step 412 of constraints generation, based on the definedconstraints.

FIG. 8 illustrates an example of converting the source code 502 intoconstraints 802, including well-formedness judgements and subtypingjudgements, at step 412 and its sub-steps 414-420, in accordance with anexample embodiment.

The constraints 802 include statements 804, 806, and 808. Statements 804are well-formedness judgement statements that define parameters a and b.

Statement 806 is a well-formedness judgement statement that defines t,which is assigned a tensor Tensor[(a,b),T]. Type parameter T may be intor float or any other suitable type.

Statements 808 are subtyping judgement statements that use sized arraysto further assess the tensor Tensor[(a,b),T] using recursion.

In some examples, well-formedness judgement statements are generatedwhen new parameters are assigned values, whereas subtyping judgementstatements are generated when primitive functions are called or invoked.

In some examples, with respect to the sized array with a formatSArray[L,T] as described above, a dependent pair is now described todefine the length of SArray[L,T] with reference to FIG. 9.

FIG. 9 shows an example of a dependent pair 902 including a firstcomponent 902(1) and second component 902(2). A type of the length L isdefined by a type of a second component 902(2) of a dependent pair 902,which depends on a value defined by a first component 902(1) of thedependent pair 902. As shown in FIG. 9, the first component 902(1)defines a range of natural numbers greater than 5. Thus, the type of thelength L is defined as a type of natural number greater than 5. Such adependent pair used in constraint generation, such as step 412 of method400, may help to acquire the data type of the length of the sized arraywith greater accuracy, which may in turn help to improve the flexibilityof the tensor representation.

When primitive functions invoke tensors as parameters in source code,the types of each size element of the tensor may have to be defined eachtime. Thus, a lot of redundant coding and excessive storage space may beneeded to define types of each size element of a tensor each time thetensor is invoked.

FIG. 10 shows an example of registering types of each size element of atensor as implicit parameters, which may help to improve the efficiencyof code and save storage space. For example, when the type declarationsof the primitive function matmul 1002 are registered, brackets “{}” areused to register types of each size element of the tensor parameters asimplicit parameters. In the example of FIG. 10, the types of the sizeelements are implicitly defined as natural numbers. In such a case, whena primitive function invoking one or more tensors is identified insource code, there is no need to define the types of each size elementof the one or more tensors each time the tensors are invoked by theprimitive function. Accordingly, computation cost and storage space maybe reduced.

With respect to generating constraint solutions (e.g., at step 422), theSMT solver may determine whether source code (e.g., source code 502) isvalid or invalid. In some examples, if the SMT solver determines thatthe validity of the source code 502 is uncertain, the verificationsystem may enter a relax mode which enables source code 502 to bemodified. This may help to double check the validity of the source code502.

FIG. 11 presents an example of adding an assert clause 1104 (e.g., anassert function) into the source code 502 to generate modified sourcecode 1102. The modified source code 1102 may help to improve theaccuracy of validity results generated by the verification system. Insome examples, it is difficult for the SMT solver to perform complicatedcomputations, such as quadratic equations. When the SMT solverencounters such computationally intensive problems, the verificationsystem may output that the validity of the source code 502 is uncertain.For example, the verification system in some embodiments may be unableto calculate an equation v=(x*(x−1))/2. Thus, the SMT solver may fail tocheck a type of a function using that equation. An assert clause orfunction 1104 may therefore be inserted for validity checking. Theassert function 1104 checks a value of an expression that is expected tobe true. If the expression is evaluated to be false, an error messagemay be displayed, or a signifier of failed verification as describedabove may be generated.

As shown in FIG. 11, in order to confirm whether a value of ret equalsx*(x−1))/2, an assert function 1104 stating “assert (ret==(x*(x−1))/2)”is added to the modified source code 1102, which may help to verifywhether ret equals x*(x−1))/2. Adding an assert function into the sourcecode 502 may help to provide a relax mode when the program is debugged,and may be applicable when tensors are used. Such a relax mode mayenable the accuracy of the validity result to be increased.

FIG. 12 illustrates an example processing system 1200 which may be usedto implement methods and systems described herein, such as theverification system implementing method 300 and/or method 400. Otherprocessing systems suitable for implementing the methods and systemsdescribed in the present disclosure may be used, which may includecomponents different from those discussed below. Although FIG. 12 showsa single instance of each component, there may be multiple instances ofeach component in the processing system 1200.

The processing system 1200 may include one or more processing devices1202, such as a processor, a microprocessor, an application-specificintegrated circuit (ASIC), a field-programmable gate array (FPGA), adedicated logic circuitry, or combinations thereof. The processingsystem 1200 may also include one or more input/output (I/O) interfaces1214, which may enable interfacing with one or more appropriate inputdevices and/or output devices (not shown). One or more of the inputdevices and/or output devices may be included as a component of theprocessing system 1200 or may be external to the processing system 1200.The processing system 1200 may include one or more network interfaces1208 for wired or wireless communication with a network. In exampleembodiments, network interfaces 1208 include one or more wirelessinterfaces such as transmitters that enable communications in a network.The network interface(s) 1208 may include interfaces for wired links(e.g., Ethernet cable) and/or wireless links (e.g., one or more radiofrequency links) for intra-network and/or inter-network communications.The network interface(s) 1208 may provide wireless communication via oneor more transmitters or transmitting antennas, one or more receivers orreceiving antennas, and various signal processing hardware and software.In this regard, some network interface(s) 1208 may include respectiveprocessing systems that are similar to processing system 1200. In thisexample, a single antenna 1216 is shown, which may serve as bothtransmitting and receiving antenna. However, in other examples there maybe separate antennas for transmitting and receiving.

The processing system 1200 may also include one or more storage units1213, which may include a mass storage unit such as a solid state drive,a hard disk drive, a magnetic disk drive and/or an optical disk drive.The processing system 1200 may include one or more memories 1210, whichmay include a volatile or non-volatile memory (e.g., a flash memory, arandom access memory (RAM), and/or a read-only memory (ROM)). Thenon-transitory memory(ies) 1210 may store instructions for execution bythe processing device(s) 1202, such as to carry out the presentdisclosure. The memory(ies) 1210 may include other softwareinstructions, such as for implementing an operating system and otherapplications/functions. In some examples, one or more data sets and/ormodule(s) may be provided by an external memory (e.g., an external drivein wired or wireless communication with the processing system 1200) ormay be provided by a transitory or non-transitory computer-readablemedium. Examples of non-transitory computer readable media include aRAM, a ROM, an erasable programmable ROM (EPROM), an electricallyerasable programmable ROM (EEPROM), a flash memory, a CD-ROM, or otherportable memory storage.

There may be a bus 1214 providing communication among components of theprocessing system 1200, including the processing device(s) 1202, I/Ointerface(s) 1204, network interface(s) 1208, storage unit(s) 1213,memory(ies) 1210. The bus 1214 may be any suitable bus architectureincluding, for example, a memory bus, a peripheral bus or a video bus.

In some examples, the processing system 1200 may be used to performmethods 300 or 400 as discussed in reference to FIGS. 3 and 4. Forexample, the processing system 1200 may register the primitive function504 wherein tensors with a format Tensor[(a,b),T] described above areapplied. The processing system 1200 may receive and compile the sourcecode 502 as described in FIG. 5B. In some other examples, the processingsystem 1200 may define well-formedness judgement constraints andsubtyping judgement constraints as illustrated in FIGS. 6A-6B and 7A-7Bfor the format Tensor[(a,b),T], in order to enable the tensors withformat Tensor[(a,b),T] to be recognized and compiled by the processingsystem 1200. Based on the defined constraints, the processing system1200 may convert the source code 502 into constraints 802 as shown inFIG. 8. As such, the processing system 1200 can recognize tensors inprimitive functions and check the validity of the source code 502,including the tensors, precisely.

In some applications, the processing system 1200 may use a dependentpair 902 as shown in FIG. 9 to define a type of a length of a sizedarray, which may help to improve the accuracy of defining the type ofthe length of the sized array. In some embodiments, the processingsystem 1200 may implicitly define types of size elements used to definetensors when registering primitive functions where the tensors areapplied, which may help reduce storage cost significantly when theprocessing system passes the size elements into the primitive functions.In some example, if the processing system 1200 outputs a determinationthat the validity of a program is uncertain, the processing system 1200may add an assert function into the program and generate constraintssolutions again, which may help to double check the uncertain resultregarding the validity of the program, which may in turn enable a resultregarding validity to be generated with greater accuracy and may enabledebugging to be performed more efficiently.

The present disclosure provides certain example algorithms andcalculations for implementing examples of the disclosed methods andsystems. However, the present disclosure is not bound by any particularalgorithm or calculation. Although the present disclosure describesmethods and processes with steps in a certain order, one or more stepsof the methods and processes may be omitted or altered as appropriate.One or more steps may take place in an order other than that in whichthey are described, as appropriate.

Through the descriptions of the preceding embodiments, the presentinvention may be implemented by using hardware only, or by usingsoftware and a necessary universal hardware platform, or by acombination of hardware and software. Based on such understandings, thetechnical solution of the present invention may be embodied in the formof a software product. The software product may be stored in anon-volatile or non-transitory storage medium, which can be a compactdisk read-only memory (CD-ROM), USB flash drive, or a hard disk. Thesoftware product includes a number of instructions that enable acomputer device (personal computer, server, or network device) toexecute the methods provided in the embodiments of the presentinvention.

Although the present invention and its advantages have been described indetail, it should be understood that various changes, substitutions andalterations can be made herein without departing from the invention asdefined by the appended claims.

Moreover, the scope of the present application is not intended to belimited to the particular embodiments of the process, machine,manufacture, composition of matter, means, methods and steps describedin the specification. As one of ordinary skill in the art will readilyappreciate from the disclosure of the present invention, processes,machines, manufacture, compositions of matter, means, methods, or steps,presently existing or later to be developed, that perform substantiallythe same function or achieve substantially the same result as thecorresponding embodiments described herein may be utilized according tothe present invention. Accordingly, the appended claims are intended toinclude within their scope such processes, machines, manufacture,compositions of matter, means, methods, or steps.

1. A method for verifying source code of a computer program written in aprogramming language, the method comprising: receiving the source code;identifying a reference to a tensor variable in the source code;compiling the source code to generate a constraint, the constraintspecifying that: the tensor variable is represented by a first sizedarray of a sized array data type, the sized array data type beingdefined by the programming language such that a sized array of the sizedarray data type comprises: a first number of array elements, the firstnumber being one or more; an associated type parameter that specifies atype for the array elements, the type parameter being selected from of aplurality of types, the plurality of types including the sized arraydata type; and an associated size parameter that specifies a size of thesized array, the size corresponding to the first number; and generatingone or more constraint solutions based on the constraint to verify thesource code by: using a satisfiability modulo theories (SMT) solver toinfer a valid typing for the source code; and determining whether thegenerated constraint satisfies the inferred valid typing.
 2. The methodof claim 1, further comprising: identifying a call to a primitivefunction in the source code; and identifying the tensor variable invokedas a parameter of the primitive function in the source code; wherein:the tensor variable is specified by the source code to include: a shapeparameter that specifies a plurality of dimensional sizes of the tensorvariable; a plurality of components, the number of componentscorresponding to the product of the dimensional sizes; and a typeparameter that specifies a type for the components; and the size of thefirst sized array corresponds to one of the dimensional sizes of thetensor variable; the associated type parameter of the first sized arrayis a sized array type; the array elements of the first sized array are afirst recursively nested set of sized arrays, such that the size of thefirst sized array and the sizes of each recursively nested set of sizedarrays within the first sized array collectively correspond to thedimensional sizes of the tensor variable; and the constraint furtherspecifies that the first sized array is input to the primitive functionto invoke the primitive function.
 3. The method of claim 2, wherein: theconstraint includes one or more well-formedness judgement statements;and generating the constraint further comprises: defining a plurality ofwell-formedness judgement constraints that state a type of the firstsized array; and generating the well-formedness judgement statementsbased on the plurality of defined well-formedness judgement constraintsuntil an associated type parameter of a recursively nested sized arrayof the first sized array corresponds to the type parameter of the tensorvariable.
 4. The method of claim 2, wherein: the constraint includes oneor more subtyping judgement statements; and generating the constraintfor the input tensor further comprises: defining a plurality of subtypejudgement constraints that state a type of the first sized array, thetype of the first sized array being a subtype of a predefined sizedarray format, the predefined sized array format comprising a sized arrayhaving a predefined type parameter value and a predefined size parametervalue for itself and each of its recursively nested sized arrays; andgenerating the subtyping judgements based on the plurality of definedsubtype judgement constraints until an associated type parameter of arecursively nested sized array of the first sized array corresponds tothe type parameter of the tensor variable.
 5. The method of claim 2,wherein: the programming language defines a dependent pair type, adependent pair of the dependent pair type comprising: a first componenthaving a value; and a second component having a type, the type beingbased on the value of the first component; and the value of the shapeparameter of the tensor variable is defined by a second component typeof a dependent pair.
 6. The method of claim 2, the method furthercomprising: registering the primitive function and the tensor variableby representing the dimensional sizes of the tensor variable as implicitparameters.
 7. The method of claim 1, the method further comprising, inresponse to determining that the source code is uncertain as a result ofverification: modifying the source code by inserting an assertionfunction into the source code to automatically verify the modifiedsource code during runtime.
 8. (canceled)
 9. The method of claim 1, themethod further comprising: outputting a result with respect to averified validity of the program source code.
 10. A non-transitorycomputer-readable medium having instructions tangibly stored thereon,wherein the instructions, when executed by a processing unit, causes theprocessing unit to perform the method of claim
 1. 11. A non-transitorycomputer-readable medium having instructions tangibly stored thereon,wherein the instructions, when executed by a processing unit, causes theprocessing unit to perform the method of claim 2.